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METHOD FOR AUTOMATIC APPLICATION AND 
MONITORING OF A STRUCTURE TO BE APPLIED ONTO A 
SUBSTRATE, AND DEVICE THEREFORE 

[0001] The present invention relates to a method for automatic application and 
monitoring of a structure to be applied onto a substrate, and a corresponding apparatus 
therefore. 

[0002] For automatic application and monitoring of a structure to be applied onto 
a substrate, it has been customary to carry out optical measurements, whereby frequently 
various systems for fully automatic testing of the structure, including adhesive and 
sealing agent lines, are used for this purpose. For this purpose, multiple video cameras 
are directed at the structure to be recognized, whereby, in addition, an illumination 
module serving to generate a contrast-rich camera image is required. 

[0003] In order to be able to monitor an adhesive line and/or adhesive trail while 
it is being applied, it is necessary to teach-in a reference adhesive trail, i.e. to have the 
camera or cameras scan the reference adhesive trail, in order to calculate therefrom 
corresponding parameters on which the assessment of the applied adhesive trails is based 
subsequently. 

[0004] However, individual components are not always supplied to the same 
position of the application facility and/or apparatus for automatic application and 
monitoring by means of the supply technology. Moreover, application of an adhesive trail 
to a fold or joining seam requires correction of the tolerances of the individual 
components and/or the position of the individual joining seams and/or folds. 

[0005] Moreover, there is a need for a method for automatic application and 
monitoring of a structure to be applied onto a substrate, preferably an adhesive agent trail 
and/or adhesive trail, whereby the application structure and/or adhesive trail is monitored 
at high accuracy while it is being applied. 

[0006] It is therefore the object of the present invention to provide a method for 
automatic application and monitoring of a structure to be applied onto a substrate, 
whereby the application structure and/or adhesive trail is monitored at high accuracy 
while it is being applied, and whereby automatic control of the application facility and/or 
positional correction with regard to positional tolerances of the individual components 
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and/or tolerances of the joining seams or similar is facilitated. 

[0007] Moreover, it is an object of the present invention to provide a suitable 
apparatus for carrying out the method according to the invention. 

[0008] These objects are met with regard to the method by the features of claims 
1 , and with regard to the apparatus by the features of claim 15. 

[0009] According to the invention, a method for automatic application and 
monitoring of an adhesive trail onto a substrate and/or component, in particular a fold or 
joining seam, is proposed, whereby a reference edge and/or reference seam is determined 
by a first camera in leading direction of the application facility, in order to control and/or 
regulate the application facility according to the reference edge by the recorded images of 
the first camera. Simultaneously and/or directly after applying the adhesive trail onto the 
substrate an/or the fold or location where components abut, a second camera carries out 
online monitoring of the applied adhesive trail in trailing direction, i.e. the adhesive trail 
is applied onto the substrate and then the second camera checks the quality of the 
adhesive trail that was just applied. According to the invention, this facilitates concurrent 
seam application guidance for two components to be glued together and online 
monitoring of adhesive application and/or sealing agent application. Thus is achieved a 
reduction of the sealing agent applied, since the seam application guidance and 
simultaneous control necessitate the use of less material due to the compensation of 
tolerances. 

[0010] Further advantageous developments are evident from the subclaims. 
Accordingly, for a three-dimensional positional correction with regard to positional 
tolerances of the 

[0011] Individual components and/or tolerances of joining seams, it is 
advantageous for the reference contour and/or a feature to be determined by at least two 
cameras, in order to carry out a three-dimensional positional correction for the 
application facility by means of the stereometry procedure. 

[0012] It is also advantageous if the two cameras record the substrate, a section of 
the component or one or more components in the form of a full image or large image, 
whereby the full images or large images of the two cameras comprise an overlapping area 
in leading direction, and whereby the three-dimensional recognition of reference contour 
position resulting in the overlapping area is used for gross adjustment of the application 
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facility prior to applying the structure. In this context, corresponding correction values 
are transmitted to the application facility and/or the robot in order to shift its coordinate 
system for the application of adhesive agent. 

[0013] If a projection is made onto the area of the reference contour for three- 
dimensional analysis, in particular if one or more laser lines are applied onto the substrate 
in the form of a projection, then a three-dimensional analysis of the profile with regard to 
the height and contour of arbitrary components can be facilitated even though this is not 
analyzable by common image processing without an additional projection. 

[0014] Moreover, it is particularly advantageous if the reference contour is 
determined just by a first camera in leading direction to regulate the progression of the 
structure to be applied according to the reference contour, and whereby the first camera 
records just a strip of the image for online regulation of the application of the adhesive 
structure. By means of this partial scan and/or partial read-out of the image recording 
chip, only small data streams need to be processed such that the image recording rate can 
be increased several-fold. In this context, the images are recorded at defined fixed time 
intervals and are independent of the speed of the application facility and/or the robot 
speed. 

[0015] If the second camera uses just a strip of the image for online monitoring of 
the applied structure, the adhesive application can proceed at high speed and the seam 
application guidance can proceed at high speed in an online fashion, since both cameras 
facilitate high-frequency image recording and rapid analysis with just one sensor with 
two cameras. In this context, a reference edge is determined in leading direction parallel 
to the online inspection of the sealing agent track applied, and the difference values are 
transmitted to the robot for correction of the track such that the accuracy of sealing agent 
application can be increased significantly and a reduction of the material needs is 
attained. Due to this only partial read-out of the image recording chip of the individual 
cameras, images of all cameras can be captured synchronously, in parallel, and 
simultaneously. 

[0016] According to an advantageous embodiment, the strips of the images of the 
cameras are recorded to form a single sequence of images and whereby the image 
recording rate is increased in line with the data reduction achieved by recording just a 
strip of the image in order to increase the speed of the automatic application and 
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monitoring of sealing agent application. By storing a single sequence of images for all 
cameras, the respective images of the individual cameras can be assigned according to the 
travel of the application facility as a function of location. 

[0017] If each camera uses only a part, in particular approx. a third, fourth or 
fifth, of the picture lines as strip of the image, the image recording rate is multiplied 
accordingly, in particular essentially three-fold, four-fold or five-fold. 

[0018] Moreover, it is advantageous for a parameterization and a recording of the 
application track to proceed in a single image recording run, whereby the images of all 
cameras are stored in a sequence of image. 

[0019] According to the invention, the stored sequence of images uses the robot 
travel path and/or the robot travel time or the robot coordinates, the position, the contrast, 
the gray scale value or color value, the width and the quality of the applied structure for 
parameterization. 

[0020] Since only a small amount of data has to be included in the calculation and 
because of the high image recording rate, it is feasible to record comparably short partial 
sections of the sealing agent application and of the reference contour and/or joining seam, 
which are, for example, between 1 mm and 3 mm, in length. Moreover, it is advantageous 
to store the structure to be applied by parameterization essentially in the form of a vector 
chain, whereby a high image recording rate and short partial sections of essentially 
between 0.5 mm and 4 mm, in particular between 1 and 3 mm, are used. The 
vectorization is advantageous in that the adhesive trail in the form of a vector chain can 
be stored in a camera-transcending global coordinate system. In contrast, traditionally, 
only a local camera image-oriented coordinate system is being used. The invention thus 
facilitates that a switch of the sensor head only necessitates recalibration and/or new 
calibration without having to teach-in the adhesive trail again. 

[0021] According to another advantageous embodiment, it is advantageous to use 
three cameras, whereby each camera is used and/or can be used both for regulation in 
leading direction according to the reference contour and for monitoring of the applied 
structure in trailing direction, whereby the three cameras each comprise an overlapping 
area to the adjacent camera on a circular line. As a result, a sensor with three cameras can 
be attached to be fixed on the application facility, since each individual camera can 
assume both the regulation of seam application guidance and online monitoring of sealing 
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agent application. Advantageously, the angle values of the circular line from 0 to 360° 
form a global coordinate system, whereby a segment of the circular line is assigned to the 
images of the individual cameras in order to carry out on this circular line either the seam 
application guidance or the monitoring of the sealing agent. As a result, two of the three 
cameras are always active for analysis, i.e. one for seam application guidance and one 
other for the monitoring of sealing agent application. 

[0022] Another advantage is that an automatic switch is made, when the reference 
contour or the adhesive trail progresses from one to the next camera, i.e. in that the 
activation is transferred from the one camera to the other camera when the application 
structure or the reference contour progresses from the segment of the circular line of the 
one camera via the overlapping area to the segment of the circular line of another camera. 

[0023] Since the images are recorded very shortly one after the other (every 0.5 to 
4, in particular 1 to 3 mm), it can be assumed that the position of the adhesive trail and/or 
joining edge cannot change too strongly which significantly improves the significance 
and/or reliability of the a-priori knowledge such that it can be predicted where the track 
will be located. One positive effect of this is that the computer can recognize fully 
automatically the position of the track even in the absence of the expert knowledge of a 
human being, since the computer already knows approximately where the track will be 
progressing in the next image. This allows the search area to be reduced and the speed of 
analysis to be increased. 

[0024] According to the present invention is provided an apparatus for automatic 
application and monitoring of a structure to be applied onto a substrate, preferably an 
adhesive line or adhesive trail, for carrying out the method according to the invention, 
whereby at least one illumination module and one sensor unit are provided, and whereby 
the sensor unit is made up of at least two cameras that are provided around an application 
facility for applying the structure to be applied onto the substrate and are arranged on this 
facility such that at least one camera is provided in leading direction for regulation of the 
application facility by means of a reference contour and at least one camera is provided in 
trailing direction for simultaneous online monitoring of the structure applied onto the 
substrate. The apparatus according to the invention can therefore be used, for example, to 
guide a seam as reference contour for the control of the application facility and/or robot 
control, and simultaneously carry out an online control of sealing agent application, such 
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that less material is used in sealing agent application since the width of the adhesive 
agent track can be reduced due to the guidance of the application facility. 

[0025] If the optical axes of the individual cameras essentially intersect, in the 
direction of view, the axial longitudinal axis of the application facility or if the optical 
axes of the individual cameras are directed to be parallel to each other, and in particular 
are directed to be perpendicular to the substrate, it is advantageous according to a 
development of this type that a narrow area around the application facility can be 
monitored at suitable resolution and high image recording rate. 

[0026] According to a preferred embodiment, the individual cameras, in 
particular 3 cameras, are arranged at equal distances from each other in the direction of 
the circumference. 

[0027] Advantageously, the individual cameras interact with each other such that 
the images of the cameras are stored in a sequence of images, whereby these images are 
composed by the software from the three partial sections of the individual cameras that 
were recorded synchronously and captured in parallel. 

[0028] If a projection facility projecting one or more features, in particular strips, 
onto the substrate for the three-dimensional analysis is provided on the application 
facility, arbitrary components can be used for correction and/or adjustment of the 
application facility prior to applying the structure. 

[0029] According to a preferred embodiment, the projection facility emits one or 
more laser lines for three-dimensional profile analysis. Arranging at least two projection 
facilities around the application facility facilitates gap-free three-dimensional analysis 
around the application facility, whereby the analysis of sealing agent height and sealing 
agent contour as well as position and width can be carried out according to the principle 
of triangulation by means of image processing. 

[0030] According to a development of an invention, the cameras are arranged 
around the application facility such that at least an essentially circular edge scan, in 
particular in the form of a circular caliper, is formed whose center is formed by the 
application facility of the structure. In this context, one or more circular calipers can be 
used that facilitate the determination of the edge of the adhesive trail on a circular line. 

[0031] According to a preferred embodiment, the individual cameras comprise an 
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overlapping area of 30° to 90° each, in particular essentially 60°, relative to the next 
camera. This overlapping area facilitates fully automatic switching between neighboring 
cameras when the adhesive trail progresses from the monitoring area of one camera to the 
next, since the selection of the camera is not bound to the robot position or a time 
component, but rather always refers to the actual inspection results, i.e. is based on the 
arrangement on the circular line of the circular caliper and/or the global coordinate 
system formed thereby. 

[0032] Moreover, it is advantageous for the illumination module to be made up of 
LEDs, in particular infrared LEDs, UV LEDs or RGB LEDs. 

[0033] Moreover, it is of advantage to use a calibrating disc with individual form 
elements for calibrating the individual cameras for the assignment of the angle 
assignment, whereby said form elements comprise, in particular, an angle distance of 
essentially 10°. This allows for assignment of the scaling factor, angle assignment, and 
center as well as radius of the search circle for the individual cameras. According to the 
invention, the calibrating disc comprises at least three marker sites that are arranged in a 
circular arc of the calibrating disc of essentially 0°, 120°, and 240°, in order to calibrate 
three cameras. 

[0034] Further advantageous developments of the invention are the subject of the 
remaining subclaims. 

[0035] Advantageous developments of the invention shall be illustrated in an 
exemplary fashion by means of the following drawings. 

[0036] Figure 1 shows a schematic side view of an apparatus according to the 
invention for application and monitoring of an adhesive trail. 

[0037] Figure 2 shows a perspective view of the apparatus according to the 
invention of figure 1 . 

[0038] Figure 3 shows the travel path of the apparatus according to the invention 
for application and monitoring of an adhesive trail. 

[0039] Figure 4 shows another travel path of the apparatus according to the 
invention with regard to the switching of the relevant camera. 

[0040] Figure 5 is a view of a single image composed from three image strips 
from three cameras for seam application guidance and online monitoring of sealing agent 
application at one edge of a component. 
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[0041] Figure 6 is another view of a single image composed from three image 
strips from three cameras, whereby two overlapping components are being glued 
together. 

[0042] Figure 7 shows a schematic view of a calibrating device according to the 
invention for calibrating the individual cameras of the apparatus according to the 
invention for automatic application and monitoring of a structure to be applied onto a 
substrate. 

[0043] Figure 8 shows a top view with regard to the basic principle of seam 

tracing. 

[0044] Figure 9 shows a top view with regard to the principle of 3D positional 
recognition. 

[0045] Figure 10 shows a top view with regard to profile analysis. 

[0046] Figure 11 is a schematic side view of the apparatus according to the 
invention with projection facility. 

[0047] Figure 12 is a schematic top view of a projection applied to be circular. 

[0048] In the following, the design of the apparatus according to the invention for 
recognizing a structure to be applied onto a substrate is illustrated according to figures 1 
and 2. 

[0049] Reference number 10 indicates the schematically shown apparatus for 
application and monitoring of an adhesive trail. In the center of the apparatus according 
to the invention is arranged an application facility 1 1 by means of which an adhesive trail 
20 is applied onto a substrate and/or onto a sheet of metal 30 proceeding from right to left 
in fig. 1. Three cameras 12, 13, 14 are arranged at equal distances from each other in a 
circle around the application facility 11, each of which is directed at the application 
facility 11. As is evident from figure 1, the axial longitudinal axes of the three cameras 
12, 13, 14 intersect the axial longitudinal axis of the application facility 1 1 just below the 
substrate 30 such that the focus of the individual cameras is arranged right around the 
area of the application facility 1 1, in particular on a circular line. 

[0050] I 
n the inspection of the adhesive, either the application facility with the cameras or the 
substrate is moved, whereby the adhesive trail 20 is simultaneously applied to the 

8 



Atty. Dkt. No. QU01H10/P-WO1/US (22589-13US) 



substrate 30 by means of the application facility 1 1, and whereby the cameras 12, 13, 14 
monitor the applied structure. For this purpose, it is feasible to move either the 
application facility with the cameras or the substrate in order to apply the adhesive trail 
onto the substrate 30 such as to follow a desired progression. In the method according to 
the invention, a first camera then determines a reference contour and/or reference line or 
reference edge in leading direction, by camera 14 towards the left in the case shown, in 
order to regulate the progression of the structure to be applied according to the reference 
contour, whereby the images recorded by the first camera are used to guide the 
application facility 1 1 in the application of the adhesive trail. The adhesive trail is applied 
by the application facility 1 1 simultaneous to the determination of the reference contour, 
whereby the application facility 11 is moved to the corresponding track and/or the 
corresponding adhesive trail progression according to the correction values determined 
by the first camera. Synchronous to this process, the adhesive application track is 
monitored by a second camera in trailing direction. By this means, the cameras that are 
being moved along can control the adhesive trail, while it is being applied, according to 
the reference contour independent of the travel path and monitor the quality of the 
adhesive trail online. In fig. 2, the adhesive trail 20 progresses from left to right which is 
shown as a continuous line. The desired progression of the adhesive trail 20 that can be 
applied to the metal sheet and/or substrate as reference contour (for example by laser or 
embossed) is shown to the right of the application facility 1 1 by means of a dashed line. 

[0051] Figure 3 then shows the progression of the adhesive trail 20 as indicated 
by arrows, whereby the direction and/or field of view of the three individual cameras is 
shown in three sites. The field of view of the three individual cameras each is indicated 
by a rectangle drawn with a continuous line, a rectangle drawn with widely dashed lines, 
and a rectangle drawn with narrow dashed lines. As is evident from figure 3, the direction 
of the individual fields of view of the cameras remains constant at all times whereby only 
the whole apparatus is moved. 

[0052] Figure 4 shows another progression of an adhesive trail 20, whereby it is 
indicated in each case, which field of view is active, i.e. which camera having the 
corresponding field of view shown as a rectangle is active while traveling along the 
adhesive trail for quality control purposes. Moreover, at least a second camera is active in 
leading direction for seam application guidance and/or the progression of the sealing 
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agent track, though this is not shown here for reasons of clarity of presentation. However, 
it is evident that the cameras indicated to be active are arranged in trailing direction and 
are switched automatically according to the progression of the adhesive trail. Therefore, 
at least one camera is active at any time for seam application guidance and fine 
adjustment of the application facility according to the given reference contour in the 
leading direction, which is opposite to the tailing direction. 

[0053] Figure 5 then shows three image strips which each represent a relevant 
section and/or strip of image of the three individual cameras of figure 1 . According to the 
method of the invention, each camera records just a strip of the image in order to reduce 
the amount of data accordingly such that the recording rate can be increased. These 
individual image strips of the three cameras are then joined into an image, whereby the 
image recording occurs at defined fixed time intervals and independent of the robot 
control of the application facility. For example, the cameras only record a strip of the 
image, whereby instead of an image height of 480 pixels an image height of approx. 100 
pixels (100 image lines) is used. By means of this partial scanning technique, i.e. partial 
reading-out of the image recording chip, only small data streams are generated such that 
the image recording rate can be increased several-fold accordingly. Synchronous image 
recording and parallel image capture allow the three image strips, one below the other, to 
be composed into a single image. As a result, the three images, i.e. the three image strips, 
are correctly arranged and assigned with regard to location and time relative to the travel 
path of the application facility without further ado, and can be processed accordingly. 
This specific image recording technique therefore facilitates simultaneous and parallel 
recording of individual camera images, which achieves an increase in the image 
recording rate both for guidance and/or regulation of the application facility and for 
online monitoring of the adhesive agent applied, whereby the images of all cameras are 
stored in a sequence of images. 

[0054] Once the images of the three cameras are stored in a sequence of images, a 
parameterization of this reference track is carried out as the subsequent step of teaching- 
in the reference adhesive trail. The robot travel path, robot travel time, direction, width, 
and quality of the adhesive trail are used in the parameterization. This results in a type of 
vector chain for the adhesive trail which allows to attain the high image recording rate 
and comparably short partial sections (between 1 and 3 mm). Vectorization has another 
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advantage in that the adhesive trail, being in the form of a vector chain, can be stored in a 
camera-transcending global coordinate system. 

[0055] As is evident from figure 5, which shows a seam application guidance at 
an edge of a component, the seam inspection is carried out online in the middle strip of 
figure 5, whereby the segment of a circle is the area, in which the first camera provides 
for monitoring of the adhesive. 

[0056] Application facilities and/or robots work with an internal interpolation 
clock time of 12 ms, for example. The regulation of seam application guidance according 
to the lower strip of fig. 5 cannot occur more rapidly than this interpolation clock time. At 
a maximal robot travel speed of 700 mm/s, this means only that the path traveled in 12 
ms is 8.4 mm. Accordingly, if a correction value is determined at time point x, the 
correction can be made only at the next interpolation clock time, i.e. 8.4 mm after 
determination of the value. As a result, the reference edge must be captured at least 8.4 
mm ahead of the nozzle. This area is covered by the arrangement of the optical sensor 
system and correction values are made available in due time by the rapid analytical cycle 
(<5 ms). Since the cameras are attached around the application facility such as to be fixed 
in position, the progression of the adhesive trail changes, whereby the seam correction 
can therefore be carried out in the first strip according to the first camera in the second 
strip according to the second camera or in the third strip according to the third camera. 
Consequently, as described above with regard to the online monitoring of adhesive 
application, another camera becomes active for seam correction when the reference seam 
migrates from the field of view of one camera into the field of view of another camera. 

[0057] The bottom strip of fig. 5 shows a bright cross on the line perpendicular to 
the edge of the component right at the edge of the component that is used as reference 
edge for seam application guidance. Paralleling this, the seam inspection for monitoring 
of sealing agent application is carried out in an online fashion in the middle strip of fig. 5. 

[0058] If the adhesive trail progresses out of the field of view of a camera, the 
adhesive trail is transiently in the overlapping area of the ranges of angles of the two 
cameras. If the adhesive trail then progresses from the segment of the circular line of the 
one camera via the overlapping area to the segment of the circular line of another camera, 
an automatic switch is made from the one to the other camera. This is shown, in 
particular, in figure 4 by means of the active fields of view of the individual cameras. 
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[0059] The advantages mentioned above are attained by the individual cameras 
forming a circular caliper whose center is formed by the application facility 1 1 , whereby 
the search for both the reference edge and the edges of the adhesive trail proceeds on a 
circular line directly around the application facility. For this purpose, it is essential that 
the individual cameras are directed at the application facility, whereby the axial 
longitudinal axes of the individual cameras intersect the longitudinal axis of the 
application facility. 

[0060] The illumination module (not shown here) for the apparatus according to 
the invention is made up of LEDs, in particular infrared LEDs, UV LEDs or RGB LEDs. 
In order to attain as little movement blur as possible and/or high contrast in image 
recording, the LEDs can be flashed, i.e. short, strong pulses of current on the order of 1.0 
to 0.01 ms can be applied to the diodes. In this context, light-emitting diodes capable of 
emitting light of various colors are particularly advantageous such that the sensor design 
can be switched to other types of adhesive and/or colors of adhesives without 
reconfiguration. 

[0061] A teach-in run and/or a teach-in of a reference adhesive trail is illustrated 
in the following. 

[0062] The teach-in process of the reference adhesive trail can be started by the 
user by marking the position of the adhesive trail. This is sufficient for fully automatic 
recognition of position and direction of the adhesive trail in the subsequent camera 
images, since the image recording rate is sufficiently high and the individual images are 
recorded very shortly after one another, for example every 1 mm to 3 mm. From the 
starting point, the adhesive is scanned image by image, whereby the adhesive trail 
position and the adhesive trail angle detected in the current image are used for the 
upcoming image as a priori knowledge. This facilitates fully automatic capture of the 
adhesive trail without a human being having to determine and/or assess the image and/or 
the position of the adhesive trail. As a result, the search area can be limited and/or 
adjusted. 

[0063] Figure 6 shows a guidance of seam application in the overlapping area of 
two components, in particular at a place at which two components abut. The second 
camera shows the strip of the second camera read-out according to the partial scanning 
method, in which the position of the overlap of the two metal sheets is determined as 
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reference contour and/or reference edge to guide the seam application. The strip of the 
third camera, in which the applied sealing agent track is monitored in parallel to seam 
application guidance, is shown in the bottom strip of fig. 6. For this purpose, the segment 
of a circle is shown in the bottom strip, in the middle of which progresses the adhesive 
trail as indicated by a circle. The image recording strip of the first camera is shown in the 
top strip of fig. 6. 

[0064] Figure 7 shows a calibration facility 40 in the form of a circular calibrating 
disc in order to assign to the individual cameras their scaling factor, their angle 
assignment, and the center as well as the radius of the search circle. The calibrating disc 
consists of individual form elements and/or dots 41 that are arranged on a circular line 
and at an angle distance of essentially 10°. Moreover, marker sites 42 are arranged at 
equal distance from each other in order to calibrate three cameras. A compensation 
calculation is used to calculate from the coordinates of the centers of the individual dots, 
on the one hand, the scaling factors of the individual cameras and, on the other hand, the 
center as well as radius of the search area. The marker sites at angles of 0°, 120°, 240° in 
the global coordinate system allow the angle assignment and the corresponding fields of 
view of the individual cameras to be determined. The field of view of the individual 
cameras is indicated, in particular, by the three rectangles in figure 7, whereby the form 
elements 41 can correspond to the circular line of the circular caliper for detection of the 
adhesive trail. 

[0065] Figure 8 shows three strips around the application facility 1 1 each by 
dashed lines that represent the read-out area for the partial scan of the individual cameras. 
The strip 31 of the first camera determines the reference edge 35 in order to control 
and/or regulate the application facility according to the progression of the reference edge. 
Image strip 3 1 is therefore facing in leading direction and measures the position of the 
reference edge and/or fold 35 such that the application facility 11 applies the sealing 
agent onto the track corrected according to the reference contour 35. After correction of 
the robot track with regard to the coach work position, the joining seam is recognized by 
driving to the first position and activating the seam application guidance. After release of 
the process (seam is recognized), the robot track continuously receives correction values 
that are perpendicular to the application direction taught-in. In this context, the capture 
area can be ± 15mm and whereby the regulation area is < ± 1 mm. The communication 
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between the image processing system and the robot system and/or application facility 
proceeds, for example, by means of a standardized Ethernet interface using an XML 
protocol. In trailing direction, the two image strips 32 and 33 are shown that intersect in 
the area of the sealing agent track 20. 

[0066] The online monitoring of an applied adhesive trail shall be illustrated 
briefly in the following. The application facility 1 1 shown in figure 1 applies the adhesive 
trail onto the metal sheet 30, whereby the application facility 1 1 is moved jointly with the 
cameras over the metal sheet 30 and regulated according to the reference contour. 
However, a kinematic inversion is also feasible, i.e. the metal sheet 30 being moved and 
the application facility with the cameras being arranged to be fixed in position. The 
applied adhesive trail 20 is determined and analyzed synchronously and in parallel by one 
of the cameras 12, 13, 14 on the circular line of the circular caliper illustrated according 
to figure 5, whereby each camera records only a strip of the image and joins these into a 
single image. The image recording rate is increased in accordance with the data reduction 
attained by each camera recording only a strip of the image, whereby the individual 
image strips in the joint image facilitate the synchronous and parallel as well as 
simultaneous capture of the three camera images, and whereby the individual images of 
the three cameras can be assigned directly as a function of location. As a result, seam 
application guidance and online monitoring of the adhesive trail in real-time is feasible 
that achieves high accuracy at high travel speeds due to the high image recording rate 
both in the regulation according to the reference edge and in the inspection of the applied 
adhesive trail. 

[0067] Fig. 9 shows the basic principle of 3D positional recognition which is 
carried out prior to applying sealing agent. Since the metal sheets, for example raw 
coachwork of vehicles, are not always positioned in exactly the same position by the 
supply technology and the position of the joining seams is associated with tolerances, 
gross adjustment and/or gross positioning of the apparatus according to the invention is 
advantageous, For this purpose, the camera image fields are switched to large image 
and/or standard size or full image, which is indicated for each case by the dashed line(s) 
51 and/or 52. The standard camera image field 51 shows the expanded field of view of a 
camera reading-out just the corresponding strip 31 according to the partial scanning 
procedure. The strip 32 is scaled down analogous to this and according to the ratio to the 
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standard camera image field 52. For example, the image strip 3 1 and/or 32 is scaled down 
by a suitable software to, for example, half its width and 1/5 of its height. For reasons of 
clarity of presentation, the corresponding standard camera image field 53 with 
corresponding image strip 33 is not shown here. In the 3D positional recognition, an 
arbitrary feature 60 within the overlapping field of view of the two camera image fields 
51 and 52 is measured. Since the two camera image fields 51 and 52 overlap in the area 
of feature 60, the procedure of stereometry can be used in order to facilitate a three- 
dimensional analysis, for example of a hole or an edge of the component. If, for example, 
a seam of two components has been recognized, the application facility can carry out an 
automatic correction of position through the help of the sensory system in order to carry 
out the correction of the robot track and/or application facility track in an online fashion, 
as described in fig. 8. The same sensory system can carry out at the same time the quality 
control of seam sealing in an online fashion, as is also described in fig. 8. This provides 
for positional correction of the application facility and online regulation of the 
progression track of the application facility as well as online monitoring of sealing agent 
application with a single sensory system that is made up, for example, of three cameras 
arranged around the application facility such as to be fixed in position. In this context, 
just a strip of the image 31, 32, 33 is recorded according to the invention in order to 
utilize small data streams to achieve an increase in the image recording rate. The use of 
the partial scanning technique thus provides for an image refresh rate of approx. 240 Hz 
or lower. The images are therefore recorded at defined fixed time intervals and are 
independent of the speed of the robot and/or application facility. In addition, a frame 
grabber board (= PC board for capturing images from the camera) is used in the 
analytical PC allowing images from all three cameras to be captured synchronously and 
in parallel. The images are subsequently composed into one image (3 strips one below the 
other) providing the advantage of three images each being directly assigned accordingly 
in a location-dependent fashion. 

[0068] Moreover, it is sufficient, in particular, to search for and analyze, for 
example, the sealing agent track in one of the three images. If the angle value exceeds a 
certain value, an automatic switch to the neighboring camera is made. In this context, the 
angle value refers to a full circle of 360°, which results in a global coordinate system. 
Each camera comprises an area of overlap with the next camera. The selection of the 
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camera is made independent of the position of the application facility and/or robot 
positron and/or independent of a time component, but rather always refers to the actual 
inspection results which are captured in the global coordinate system. This prevents 
errors that are generated by the relative inaccurate robot controls and/or application 
facility controls. 

[0069] The three-dimensional profile analysis by means of a projection is 
described according to figures 10 and 1 1 5 in order to provide for a positional correction of 
the application facility as described above according to figure 9. For reasons of clarity of 
presentation, fig. 10 again shows only two camera fields of view 51, 52 indicated by 
dashed lines. In the overlapping area of the two camera fields of view 51, 52 are shown a 
plurality of laser lines 60 that are used for profile analysis with regard to the width and 
contour of structure lines and for generation of so-called soft contours. The laser lines 60 
are generated by a projection facility that can, for example, be arranged on the optical 
sensor with three cameras. Moreover, the projection facility can just as well be arranged 
directly on the application facility 11. The sensor with the three cameras is shown 
schematically by the circle 70. The laser lines and/or laser strips projected onto the 
component 30 and/or metal sheet 30 highlight contours on the component that cannot be 
used for three-dimensional analysis by conventional image processing. Artificial features 
are generated by means of the laser lines on the component and can subsequently be 
analyzed by means of image processing according to stereometry. Thus, figure 10 shows 
the principle of three-dimensional positional recognition prior to the application of 
sealing agent in case no hard, analyzable features are present. In contrast, a hard contour 
is described by means of feature 60 in fig. 9 as mentioned above. 

[0070] Fig. 1 1 shows a side view of the application facility 1 1 with sensor unit 70 
attached to it, whereby the sensor 70, aside from the three cameras, can comprise at least 
two projection facilities 61 that project laser lines onto the metal sheet and/or the 
substrate 30, as is shown schematically by the dashed line. Arranging multiple projection 
facilities 61 around the application facility allows a gap-free contour to be generated on 
the metal sheet 30, whereby the gap-free contour can be used for three-dimensional 
analysis since the sensor and the projection facility are calibrated. Accordingly, fig. 1 1 
shows two projection facilities 61 in an exemplary fashion. Projection facilities of this 
type can, for example, project a laser onto the substrate and/or component or can be made 
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up of an LED module comprising an adapter lens to generate a line on the substrate. 

[0071] The projection facilities can be used both for three-dimensional positional 
correction prior to sealing agent application as well as for online analysis of height and 
profile of the applied sealing agent. For three-dimensional positional correction, the 
projection facilities can preferably project multiple lines. For height analysis, one or more 
projection facilities should be provided that project a line or, as shown in fig. 12, a 
circular contour onto the component and/or substrate. In this context, multiple lines can 
be advantageous for the analysis. 

[0072] As shown in fig. 12, it is also feasible to determine sealing agent height 
and/or sealing agent contour and sealing agent position according to the principle of 
triangulation by means of the image processing simultaneously and directly after sealing 
agent application. For this purpose, a round contour 63, for example, is applied to the 
metal sheet 30 by the projection facilities, whereby the sealing agent and/or the sealing 
agent track 20 provides for a change of height and position of the projected contour 63. 
The round projection contour 63 thus changed in turn is determined by the individual 
object fields of the individual cameras. The original shape of the projected contour 63 is 
thus deformed by the sealing agent 20 such that the width, contour, but also the height 
and position of the applied sealing agent 20 can be determined according to the principle 
of triangulation. In the principle of triangulation, there is a defined angle between the 
camera and the projection facility, whereby the camera and the projection facility are 
calibrated with respect to each other. Because of the angle, the substrate contours 
illuminated by the projection facility appear in different positions on the light-sensitive 
chip and/or CCD chip or CMOS chip as a function of their height such that the height and 
contour of the sealing agent can be calculated due to the calibration of the camera and 
projection facility. 

[0073] According to an embodiment that is not shown here, the sensor that is 
made up, in particular, of three cameras and is arranged around the application facility is 
made up such that the optical axes of the individual cameras are directed such as to be 
parallel to each other, whereby the cameras are, in particular, directed such as to be 
perpendicular to the substrate and/or metal sheet. An arrangement of this type allows the 
sensor to be arranged particularly close to the area of sealing agent application, whereby 
the fields of view of the individual cameras comprise an overlapping area whose size 
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